Field of the Invention
The present invention relates to an image processing method that is capable of performing a reconstruction for an input image so as to generate a plurality of output images that have focus positions different from each other.
Description of the Related Art
Recently, an image pickup apparatus that performs a calculation using data obtained by an image pickup element and performs a corresponding digital image processing so as to output various images is proposed. Ren Ng, et al., “Light Field Photography with a Hand-held Plenoptic Camera”, 2005 Computer Science Technical Report CTSR, and Todor Georgiev, et al., “Superresolution with Plenoptic 2.0 Camera”, 2009 Optical Society of America, disclose an image pickup apparatus that simultaneously obtains the two-dimensional light intensity distribution and parallax information (collectively, referred to as a “light field”) on an object plane. According to this image pickup apparatus, obtaining the light field, a focus position of an image can be changed by an image processing after taking an image, which is called a refocus, and a depth of field can be adjusted. However, in this image pickup apparatus, pixels of the image pickup element need to be allocated to store the parallax information, as well as the two-dimensional light intensity distribution of the light. Therefore, a spatial resolution is deteriorated compared to an image pickup apparatus which stores only the two-dimensional light intensity distribution of the light.
Todor Georgiev, et al., “Superresolution with Plenoptic 2.0 Camera”, 2009 Optical Society of America, discloses a configuration in which a certain point on an image plane that is formed by an imaging optical system is taken by a plurality of small lenses constituting a lens array. A plurality of small images obtained like this are reconstructed so that the resolution of the reconstructed image can be improved. Such a method of improving the resolution is referred to as a “super-resolution from subpixel shift”. Todor Georgiev, et al., “Superresolution with Plenoptic 2.0 Camera”, 2009 Optical Society of America, discloses a method of obtaining an effect of the super-resolution from subpixel shift at a specific focus position. However, it does not consider how to change the effect of the super-resolution from subpixel shift when an image is generated at a plurality of focus position different from each other. Therefore, a focus position at which a refocus image has a high resolution cannot be specified. If the effect of the super-resolution from subpixel shift at the plurality of focus positions is calculated, a high-resolution refocus image can be automatically generated by using its result, or a user can be informed of a focus position at which the great effect of the super-resolution from subpixel shift is obtained.
Furthermore, Todor Georgiev, et al., “Superresolution with Plenoptic 2.0 Camera”, 2009 Optical Society of America, discloses a method of obtaining the effect of the super-resolution from subpixel shift at a specific focus position. However, an effect of the super-resolution from subpixel shift when generating a refocus image at an arbitrary focus position is not disclosed. Since the effect of the super-resolution from subpixel shift changes in accordance with the focus position, there is a case where a focus position with high resolution may exist within a range capable of refocusing compared to a specific focus position. Thus, an object designated by a user at the time of taking an image (a main object) does not necessarily exist on a focus position with high resolution.